Estimator Variance in Reinforcement Learning: Theoretical Problems and Practical Solutions

In reinforcement learning as in many on line search techniques a large number of estimation parameters e g Q value estimates for step Q learning are maintained and dynamically updated as in formation comes to hand during the learning process Excessive variance of these estimators can be problematic resulting in uneven or unstable learning or even making e ective learning impossible Estimator variance is usually managed only indirectly by selecting global learning algorithm parameters e g for TD based meth ods that are a compromise between an acceptable level of estimator perturbation and other desirable system attributes such as reduced estimator bias In this paper we argue that this approach may not always be adequate particularly for noisy and non Markovian domains and present a direct approach to managing estimator vari ance the new ccBeta algorithm Empirical results in an autonomous robotics domain are also presented showing improved performance using the ccBeta method